Eduardo Gonzalez

notes

Software 2.0

Andrej Karpathy, 2017-11-11

  • Karpathy's thoughts on neural networks as a shift in software developement, from traditionally writing explicit coding to curating large datasets and optimizing neural architectures + neural nets and ml models (potentially democratizing programming). i think this 2017 (!) essay is fundamental in understanding the shift we're going through (2016-?) and its implications on how we interact with code, computers and technology.

⠀

  • Software 1.0: "classical stack" - Python, C++
    • explicit instructions to the computer by a programmer, who identifies a specific point in program space with some desirable behavior.
    • human engineered code (.cpp files) is compiled into a binary.
  • Software 2.0: database + neural net architecture
    • abstract, human unfriendly.
    • No human involved in writing code since weights are the main part (typical networks have millions)
    • code = dataset that defines desirable behavior + neural net architecture (skeleton of code) with many details (weights) to be filled in.
    1. specify some goal (e.g. win a game of go) on the behavior of a desirable program
    2. write a skeleton of the code (i.e. neural net architecture) that identifies a subset of program space to search
    3. use computational resources to search this space for a program that works
  • significantly easier to collect data/identify a desirable behavior than to explicitly write a program with real-world problems
  • neural nets/training systems are increasingly standardized into a commodity
    • ⤷ today's active "software developement" is curating/cleaning/growing labeled datasets.
    • e.g. Speech recognition: preprocessing + gaussian mixture models + hidden markov models → now ~100% neural nets
  • Software (1.0) is eating the world, and now AI (Software 2.0) is eating software.

note mentions

Eduardo Gonzalez

ego